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ABSTRACT 



It is the function of this paper to 
specify, both by verbal and figurative represen- 
tation (in the form of a model) , the nature of 
what this author feels should be observed and 
judged in evaluating instructional programs. 

Evaluation is defined as "The process 
of first identifying and quantifying the relation- 
ships between student inputs and educational out- 
puts, and determining the combination of mediating 
factors which maximizes the educational outputs, 
given a constant financial input and controlling 
for the effects of external systems." The paper 
is primarily a discussion and amplification of the 
above definition. 
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TOWARDS AN EVALUATION MODEL: 

A SYSTEMS APPROACH 

Marvin C. Alkin 

Evaluation is the process o£ first identifying and then 
quantifying, or measuring, the relationships between student 
inputs and educational outputs and determining the combina- 
tion of mediating factors which maximizes the educational 
outputs, given a constant financial input and controlling for 
the effects of external systems. Evaluation is a complex 
activity that involves the identification of many factors 
that contribute to educational outputs. 

In our definition of evaluation, "student inputs" re- 
fers to the nature and characteristics of the students enter- 
ing the program to be evaluated. By "educational outputs" we 
mean two things: 1) cognitive and non-cognitive changes which 

take place in students after they are exposed to the instruction- 
al program. These changes are assumed to be attributable to the 
program. And, 2) the impact of the program upon systems ex- 
ternal to it (home, community, other programs, etc.). "Finan- 
cial inputs" refers to the financial resources made available 
for carrying on the program, "Mediating factors" are the de- 
scriptive characteristics (e.g., personnel, school organiza- 
tion and programs, and instructional design) of the way in 
which financial inputs are utilized within the program in 
combination with the student inputs. And, finally, by "external 
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systems’’we mean the framework of social, political, legal, 
economic, and other systems outside of the school, formal or 
informal, which encompass the program, have impact upon it, 
and are, in turn, modified by the outputs of the program. 

It should be specifically noted that in the discussion 
of mediating variables, we act under the assumption that they 
are the only set of variables which are manipulatable . For 
the sake of this model we will assume that: 1) external 

systems are not immediately altered by the outputs of the 
system; and 2) that the school decision-makers have no contr,ol 
over which external systems are allowed to impinge upon the 
school. If we were to maintain that feedback immediately 
changes the system, this would imply a highly dynamic model 
rather than the static model considered here. The second 
assumption implies that no attempt will be made to change the 
nature of the student inputs to the system--that is, we do not 
usually concern ourselves with the consideration of community 
changes, which might be made, that would alter the nature of 
the student inputs. We act, too, under the assumption that 
student inputs are relatively nonmanipulatable from without 
the system. Thus, we concern ourselves with the mediating 
variables within the system that can be changed, manipulated 
and altered in order to maximize student outputs. We recognize 
the weakness in this assumption, and that there are some 
school-related manipulations that could be instituted which 
v;ould change the nature of the student input. Instances of 
this are bussing, changing of school boundaries in order to 
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"juggle" student inputs to specific schools, community educa- 
tional resources (such as education resource units in disad- 
vantaged areas), and preschool programs (such as Project 
Headstart) . The assumptions of a static model and of non- 
manipulatable external systems seem necessary at this early 
stage of the model development. 

With our definition of evaluation and some of the limits 
we are imposing in mind, it seems appropriate to discuss the 
evaluation model^ and, in more general terms, the principles 
of evaluation which follow from the model. However, before 
doing this, we will first discuss the general nature and func- 
tion of an evaluation model and, second, the importance of the 
size of the unit to be evaluated upon the nature of the eval- 
uation. These sections will set the background for the 
development of the model presented in this paper. 

It should be noted that in this paper we are not pri- 
marily concerned with the methodological consideration per se . 

We accept the differentiation by Stake (1967) between the theory 
of evaluation which identifies what is to be observed and 
judged and the methodology of evaluation which specifies the 
manner in which these observations and judgments will be made. 

It is the function of this paper merely to specify, both by j 

figurative and verbal representation (in the form of a model) , \ 

the nature of what should be observed and judged. 

An evaluation model or, for that matter, any model is a 
simplistic statement or representation of sets of complex 
interrelationships. Such a representation is intended to help 
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the modelers in structuring the universe, or that segment of 
the universe being considered. Of necessity, models must be 
functions of the frames of reference of their builders. Thus, 
an evaluation model constructed by one whose background is 
psychology and the study of tests and measurements will be a 
quite different model from that constructed by a sociologist 
looking at evaluation, or of that constructed by a school ad- 
ministrator or an economist. While each model may view the 
same set of interrelationships, it is inevitable that certain 
concerns of the model builders will differ. The resultant 
generation of different perspectives of the problem is bene- 
ficial, not only for the model builder who is better able to 
"get a handle on" his universe, but to others who may be 
forced for the first time to consider more seriously some of 
what they previously may have considered to be lesser con- 
cerns in their own model. 

We cannot claim that the elements of the model presented 
here are uniquely different from those mentioned by others con- 
cerned with evaluation. In fact, most of the variables and 
kinds of interactions have been mentioned previously by 
Stake (1967), Scriven (1967), and, we are sure, many others. 
Moreover, the model is in many ways not significantly differ- 
ent from the total model of evaluation that guides the ef- 
forts of the Center for the Study of Evaluation of Instruc- 
tional Programs at UCLA, as developed by the committee of 
authors of the proposal for the Center: Chairman 

M.C. Wittrock and committee members Marvin C. Alkin, John 
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Bormuth, Reginald Jones, James Liesch, and David Wiley (1965). 
That model maintains that evaluation cannot take place with- 
out considering the nature of the instructional parameters 
in the program being evaluated, without understanding and 
quantifying the impact of social and administrative contexts 
that impinge upon the program, and without considering a 
multiplicity of outcome measures. This author participated 
in the development of that model, and the statement presented 
here simply represents his own personal modifications of 
that model, as well as deviations based on his own frame of 
reference . 

Units of Aggregation 

As a final preliminary, it should be noted that the 
nature of an evaluation depends, in part, upon the size of 
the unit to be evaluated- -and evaluation may involve many 
basic units of examination. Where the unit of evaluation is 
relatively small and discrete (e»g.» individual students), 
the nature of the students and of the financial resources 
made available, as well as many other variables which medi- 
ate in the production of outcomes, can be manipulated by 
the method of selection of groups. In this instance, the 
task of evaluation becomes the comparison of a single medi- 
ating variable to another variable or to several others. 

Where more complex units are to be evaluated, other factors 
may be more diffuse and uncontrollable. In these instances, 
there might be a need for a total systems analysis in which 
we attempt to apply statistical controls to many of the 
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dimensions being examined, because randomness of selection 
cannot be guaranteed in the same way as when smaller units 
are the basis of analysis. Furthermore, in order to fully 
develop an evaluation model, it is fundamental and, there- 
fore, necessary to understand the nature of interrelationships 
between variables being controlled. It is obvious, then, 
that the nature and sophistication of the evaluation model to 
be developed is dependent, in large part, upon the level of 
aggregation of the unit for evaluation. 

Let us examine several possible levels of aggregation 
for the units which might be considered in an evaluation. For 
instance, at the first level, it is possible to examine the 
performance of an individual student in a given classroom in 
a given school. This is done quite frequently when we give a 
nationally- standardized test to students and we indicate the 
percentile score of the student as compared to national norms* 
In this situation, we are appraising the performance of the 
student on the basis of national criterion dimensions. How^ 
every this is not evaluation unless there is a comparison 
measute available, such as a pre-test or a test in which com- 
parable groups ate treated differently. In other words y it 
seems that a clear distinction can and must be made between 
an appraisal that is simply a statement of the present achieve- 
ment or status of an individual and an “svaluatidn** that 
measures the changes in these individuals or other units while 
attempting to determine the set of mediating factors which is 
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most directly related to the given changes. The process of 
evaluation, then, requires a specific identification of the 
nature of the student units to be evaluated and stricter con- 
trols and understanding of the variables which mediate upon 
those units to produce the educational changes in a range of 
different kinds of educational outcomes, such as performance, 
attitudes, and social adjustment. With this specific meaning 
of the word "evaluation" in mind, let us proceed to examine 

other aggregated levels of units for evaluation. 

At a somewhat more aggregated level, it is possible to 

evaluate the performance of groups of children. For example, 
using classrooms as the unit of examination, we might ad- 
minister a given test in a specific achievement area to stu- 
dents within a school. Then we could proceed to discuss the 
variation in scores between these units, indicating the range 
of the scores, the mean, median, interquartile deviation, and 
other appropriate measures. In this case, while it is true 
that we are examining the performance of individual students, 
we are aggregating or combining their performances to repre- 
sent a larger unit. The evaluation in this instance still 
conforms to our previously-indicated definition. The stu- 
dent inputs are designated by the series of measures most 
descriptive of the total classroom as a unit, and the partic- 
ular set of mediating variables which has been acting upon 
this class or upon all of the individuals in the class also 
is indicated by various measures. 
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In the examination and comparison of several classroom 

»l 

units, if there have been different financial inputs to each j 

of the units, it is necessary to determine the exact nature of 
the financial input to each of the units. This would be a pre- ? 

liminary step in determining the manner in which these funds 
have been utilized in initiating environmental changes which 
mediate upon the student in the production of the educational 
outcomes. Finally, it is necessary to represent the educa- 
tional outcomes of each of the classroom units. It would be 
possible to compute the average of the scores made by students 
within one classroom and to compare it with the average score 
o£ pupils within another classroom or, for that matter, many 
other classrooms. These comparisons, when considered along 
with the other factors in the evaluation model, would repre- 
sent a first-step evaluation of the performance of groups of 
individuals. In this case, the evaluation would be made be- 
tween classrooms. 

As we have illustrated, there are a number of differ- 
ent kinds of evaluation that might be made. We might wish to 
evaluate the results on the students of a number of classes 
using a specific course of study. Or, we might wish to 
evaluate--in a specific aggregate sense of the words "instruc- | 

tional program"- -units which are broader than classroom units. 
Furthermore, we might wish to consider all of the curricular 
elements of the instructional program of the school, (mathe- 
matics, English, social studies, etc.), as subsets of the 
total instructional program, namely, the total field of aca- 




- 9 - 

demic experiences of an individual school. In this sense, we | 

would evaluate the instructional program of the school on a i 

1 

number of output dimensions by using the total aggregated score I 

of all students within the school and statistical measures de- 
scriptive of that data. 

In an instance where the unit for evaluation is a total 
school, it would be necessary to examine specific school 
factors which might have a relationship to the outputs- -that 
is, it would be necessary to ascertain to what extent relevant 
community characteristics differ between the schools. There 
are other questions which might be asked: Does each school 

provide the same amount of financial resources to be utilized 
in purchasing the important elements within the school system? 

How much is spent on teachers? On instructional services? 

What are the unique social characteristics on the school, etc.? 

With these somewhat tangential thoughts as a reference 
point, we will consider more directly the model being pre- 
sented. 

Towards an Evaluation Model 

In evaluating instructional programs, the complexities 
of the interrelationships are so great that we feel that the 
development of a theoretical model demands a systems approach. j 

I 

For this model, we will consider the school (individual school | 

or school district) as our unit of examination. 

There are two categories of input to the school: student 
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and financial. Moreover, there are a group of mediating 
variables within the school. Some of these variables are 
’’costly” and require the utilization of financial inputs; 
others are relatively cost free. Finally, there are a number 
of external social, political, and economic systems impinging 
upon the school. These factors taken together produce a 
number of outputs. Some of these are student outputs (such 
as changes in attitudes, skills, etc., of students); others 
are incidental or non- student outputs (such as program-caused 
or program- related changes in external systems). In suc- 
ceeding paragraphs, we will consider each of these elements 
individually. For ease of communication, we will hereafter 
refer to the school as ’’the system.” The system and its ex- 
ternal systems will be called the ’’macro- system. ” 

One final note must be added: We recognize that there 

are great overlaps between individual elements of the macro- 
system. However, to avoid confusion in the discussion of the 
model, we will think of the elements as being reasonably dis- 
crete. Thus, for example, certain aspects of the description 
of the student inputs are, to a great extent, a reflection of 
external social systems. They will, however, be considered 
in one category alone. (See Figure 1, page 12 for a simpli- 
fied diagram of the conceptual model.) 

I 

Student Inputs 

We will consider the student input as a description or 
measure of the student being introduced into the system or, 

• ' j 
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in the case of a larger unit of instructional program, as an 
aggregated, statistical description of the students being in- 
troduced into the system. In the ideal world, when students 
enter the system, they are given a complete battery of all 




the traditional kinds of achievement, intelligence, and per- 
sonality tests, as well as questionnaires and other documen- 
tary data describing their homes, status in the community, 
family background, family memberships in other social sys- 
tems, and the like. Unfortunately, the ideal world does not 
exist. We must, therefore, develop a series of proxy meas- 
ures of student inputs. Very frequently, intelligence scores 
are available for entering students; there is usually, also, 
some small amount of family data available in the cumulative 
record folder. Occasionally, achievement tests given in the pre- 
ceding year or two have been transferred and are available as 

a measure of the achievement starting-point of the students 
in the system. A considerable amount of additional desired 
data must, in consequence, either be collected in the school 
or more often be inferred from other, more accessible meas- 
ures. As a result, we often look at the community and the 
characteristics of the community as an indication of the kind 
of student input that is being introduced into the system. 

For example, from the works of McClelland (1953) and others, 
we know that certain family characteristics bear a strong re- * 
lationship to achievement motivation in children. And, while 
we would undoubtedly prefer individual achievement-motivation- 



test scores for each student entering the system, when such 
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FIGURE 1 

The Macro-System of an Evaluation Model 




*As student and non- student outputs enter the macro -system, 
they alter to some extent the nature of the external systems 
and thus by extension the inputs in succeeding stages. 
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information is unavailable, certain implications can, nonethe- 
less, be drawn from certain data items descriptive of family 
characteristics . 

Financial Inputs 

There is a second class of inputs to the system--finan- 
cial inputs. If we think of a district as a system, then not 
only do students enter the system, but finances are provided 
from local, state, and federal sources and are, in part, a 
means of implementing different sets of mediating factors 
within the system. Perhaps it is relevant to determine the 
portion of the total resources, derived from each of the gov- 
ernmental levels. Perhaps, also, it is important to designate 
the specific authorizations from federal funds or special state 
programs in order to be aware "of the strings attached" and 
consequent implications for resource utilization within the 
system. 

If we were concerned with evaluating a part of the sys- 
tem, such as the mathematics program or the guidance program, 
it would be necessary to determine the nature and amount of 
the financial input to that portion of the system. Unfortu- 
nately, present accounting practices in all states provide 
data only on functions of expenditures rather than on pro- 
grams of expenditures--i.e . , data are available on a number 
of factors such as the amount spent for administration, main- 
tenance, operation, instruction, and fixed charges, but these 
data are not available on a program basis. What is needed, 
therefore, is a budgeting system that will allow ready 
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aggregation and disaggregation of funds to provide specific 
and total cost data on individual instructional programs. In 
short, we need a system of program budgeting. 

External Systems | 

The school is placed within the framework of numerous 
social systems (external social contexts). For example, in 
the case of the individual school some of the contexts are: 
the community, the district, the nature of the district or- 
ganization, other governmental systems such as the city, the 
county, the patterns of community organizations and of commun- 
ity participation. Each of these external systems, by the 
nature of the differentiated functions it serves, places sets 
of demands and restrictions both upon the educational system 
(school) and upon the individuals within the system. Each of 
these systems serves specific integrative, adaptive, goal- 
attaining, and pattern-maintaining functions in the macro- 
system. Consequently, it is necessary to identify and quantify 
these external systems characteristics and relationships which 
are relevant in terms of the contribution they make towards 
producing the educational outputs of the system. 

In actuality, the external systems interact with the 
educational system. While each of them may be conceived as 
having their own inputs, particular sets of mediating vari- 1 

ables, and outputs, they are, in turn, external systems to the | 

educational system and vice versa . Thus, each system external 
to education may, in effect, be considered as both a source 
of inputs and a receiver of outputs. In this model, we are 
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concerned specifically with only two of the inputs to educa* 
tion, financial and student, and will consider the ’^inputs" 
from other external systems as control variables. The outputs 
of the system to external systems are considered as non- 
student outputs. 

Mediating Factors 

A fourth group of elements of the evaluation model is 
what we call the mediating factors. The financial input to a 
system can be utilized in a great number of ways. We could 

decrease the student-to-teacher ratio, establish standards 

^ # 

which insure the hiring of teachers with specified character- 
istics, develop different administrative arrangements within 
the school, provide more library books, provide more textbooks, 
introduce different curricula, use different instructional 
procedures, or provide additional supplies. 

Thus the variables that we call "mediating" are highly 
manipulatable- -that is, they are subject to change or manipu- 
lation by educational decision-makers at all levels. We have 
no definitive evidence, however, as to which combination of 
mediating variables is most effective in achieving the ob- 
jectives of the school (i.e., in producing desired educational 
outputs) . 

At this point, it is only fair to indicate that we do 
not mean to imply that all mediating factors which have impact 
on educational outputs are related to financial input. Indeed, 
some mediating factors are "free." For example, the cost of 
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implementing certain alterations in the school environment or 
in the attitudes of teachers may be relatively cost free. 
Frequently, the instructional procedure used by the teacher 
in the classroom (the substitution of one procedure for an- 
other) has little or no additional cost attached to it. How- 
ever, some changes in the system are extremely costly (such 
t as some of the administrative or organizational arrangements 
and many instructional procedures which are technologically 
based). As a consequence, the potential output achieved by 
the change must be examined in terms of the costs involved. 

It is a relatively easy position to maintain that 
more money should be provided for teacher salaries and that 
in this way, in all likelihood, the educational program will 
be improved. There is evidence that a relationship exists 
between higher teacher salaries and educational quality. 

The real question, however, is to what extent a given 
dollar input, if utilized in an alternate manner, would 
increase the nature of the educational outputs. This is a 
cost-effectiveness question and is, after all, one of the 
elements at the heart of evaluation or, at the very least, 
one of the reasons why we evaluate. 

We have noted that the selection of different sets of 
mediating factors may lead to the maximization of educational 
outputs in a system. There is, though, another point to be 
made: not only are there different sets of mediating vari- 
ables most applicable for producing given educational outputs, 
but, significantly, these sets of mediating variables may 
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produce quite different levels of change in the educational 
outputs in different systems or for different student input 
groups. James Coleman observed this point in a study for 
the Civil Rights Commission entitled Equality of Educational 
Opportunity . He noted that the "inference might then be made 
that improving the school of a minority pupil may increase 
his achievement more than would improving the school of a 
white child increase his. Similarly, the average minority 
pupil’s achievement may suffer more in a school of low 
quality than might the average white pupil’s." He concluded 
that "this indicates that it is for the most disadvantaged 
children that improvements in school quality will make the 
most difference in achievement." (Coleman, 1966). Appro- 
priate med-iating factors, therefore, are a function not 
only of the desired educational outputs, but of the nature 
of the student inputs and of the given system as well. 

As mentioned at the beginning of this paper, the 
mediating variables, as we have defined them, are assumed to 
be the only set of variables which can be manipulated. (See 
page 2.) This is a simplifying assumption, in part, because 
it allows us to deal with a static instead of a more complex 
dynamic model. Also, it should be noted that the bias implied 
by this assumption follows from the basic intent of the model 
we are seeking to construct- -that is, a decision-making model 
or, more specifically, a model designed to aid school admin- 
istrators in their day-to-day operations. 
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Student Outputs 

Another set o£ elements of the evaluation model is the 
student outputs. Changes take place in students from the time 
they enter the system to the time they leave, and many of 
these changes are produced by the nature of the mediating fac- 
tors within the system. Here, again, there is a problem, for 
the outputs of a school or of a district cannot simply be 
measured by the scores of students on academic achievement 

3 

tests. What are the noncognitive aspects of output? How 
has the behavior of students changed? What is the relation- 
ship between the activities that take place in a district or 
a school and the eventual success of students in their voca- 
tional or future educational endeavors? How does the stu- 
dent's educational experience aid him in dealing with 
political problems and activities as well as cultural affairs? 
To what extent does the social situation present in the school, 
as well as what is learned in classes, affect the student? 
These are only some of the unanswered questions related to 
the identification of educational outputs, and, of course, 
they can be solved only through further research and 
investigation. 

While there are two prime inputs into the system 

(student and financial), we will consider that there are no 

direct, financial outputs except as a portion of the student 

outputs- -there are no financial outputs except as we are 

willing to place financial value on certain behavioral changes 

or except as student outputs yield financial or economic 

4 

returns, either individual or societal. 
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Non-Student Outputs 

The final set of elements of the model is the non-stu- 
^ dent outputs. The two groups of outcome measures (student and 

non- student) may be thought of as feedback loops in which each 
modifies, to some extent, the nature of future inputs to the 
system. The changes in students, for example, have social, 
political, and economic implications- -that is, the very nature 
of the external systems is altered by changes in student out- 
puts. There are, however, other outputs of the school- -the 
impact of educational decisions made as a part of the 
"mediating factors" has repercussions in the external systems. 
Frequently, these outputs are only tangentially related to in- 
dividual students or to student outputs. For example, the 

nature of many of the decisions as to the proper utilization 

i.; 

I of resources may produce innumerable educational outputs not 

directly student-related. That is, decisions which influence 
the number and salaries of teachers, as well as the number 
and salaries of classified personnel, could, in many ways, 
modify the nature of some external systems, especially if 
these employees were to reside in the district. To what ex- 
tent do teachers paid at different salary levels have the 
economic ability to forego other earnings and instead partici- 
pate in community activities and organizations? And, how is 
the nature of these external systems modified by the educa- 
tional decision that determined the particular combination 
of mediating factors which allowed greater salaries for the 
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teachers? Also, how does the type and quality of teachers 
selected affect the changing nature of the community? Other 
examples might be the impact upon the economy of the community 
brought about by the selection of mediating factors which in- 
clude large capital investment or a large amount of supplies 
and materials locally purchased. How do the educational de- 
cisions related to whether school transportation will be pro- 
vided, or the hours of school, or the scheduling of student 
time, in terms not only of regular session classes but with 
respect to recreational and summer use of school facilities, 
have implications for parental employment patterns or 
avocational participation? And, to what extent does the 
school, as a merchant of facts, knowledge, and ideas, in- 
fluence community attitudes on political, social, and cul- 
tural issues? Finally, although the list could be extended 
greatly, how does the impact of selection of mediating fac- 
tors upon the social patterns within the school relate to 
breaking down or reinforcing patterns within the systems ex- 
ternal to the school? 

Conclusion 

We recognize that it is not possible to isolate every 
conceivable element of the total system and to determine its 
value or its individual, contributory relationship to the 
educational outputs of the system. Nevertheless, it is 
requisite in any evaluation scheme to identify and control 
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for as many of the factors as possible thought to be 
significant. It is true that in discussing each of the 
sections of this paper, we have asked many questions and, 
undoubtedly, could have asked countless more; but few 
answers have been given. This is in keeping with the gen- 
eral intent of the paper. We are primarily attempting to 
set forth the skeleton of a model that will guide our 
thinking and research in the years to come, and will prob- 
ably become vastly expanded in the process. In this case, 
we are describing what we choose to call a systems approach 
to evaluation, which has considered within it the full range 
of systemic problems including those related to input 



FOOTNOTES 



More specifically, this is a mathematical model which 
will be used in evaluating instructional programs. 

Economists would classify inputs to the system in a 
somewhat different manner: In addition to the above 

mentioned inputs, they would include teacher resources 

(including time) and student time. This kind of 
classification implies a concern for total manpower 
utilization. On the other hand, while we are con- 
cerned with such things as student time, we are 
taking as a prime consideration the decisional require 
ments of individual educational administrators at the 
local level. Thus, the model discussed is more an 
education-decision model than a total manpower model. 

We would readily admit, however, and to the chagrin 
of many reluctant school administrators, that at 
least this measure would be a feasible starting point. 



There is evidence that this is a reasonable approach. 
See: Becker (1962), Miller (1962), and Schultz (1961) 



'*■ •'v.w.w 
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